{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 基于物品的协同过滤"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用的数据集是公开音乐数据集 Million Song Dataset(MSD) ， 它 包 含 来 自 SecondHandSongs dataset 、 musiXmatch dataset、Last.fm dataset、Taste Profile subset、 thisismyjam-to-MSD mapping、tagtraum genre annotations 和 Top MAGD dataset 七个知名音乐社区的数据。 \n",
    "原始数据集包括： \n",
    "1. train_triplets.txt：三元组数据（用户、歌曲、播放次数） \n",
    "2. track_metadata.db：每个歌曲的元数据 \n",
    "由于原始数据太大，作业用的数据集只是其中的子集（播放次数最多的800个用户、播放次数最多的800首歌曲。 \n",
    "数据预处理过程请见1.DataProcessing.ipynb文件（不必运行该程序，运行该程序需要原始数据。可以通过看代码理解数据提取过程），最后得到的数据文件为：triplet_dataset_sub.csv（37000+条记录） \n",
    "\n",
    "二、作业要求：将triplet_dataset_sub.csv中的数据用train_test_split分成80%数据做训练，剩下20%数据做测试。 \n",
    "1. 实现基于用户的协同过滤； （20分） \n",
    "2. 实现基于物品的协同过滤； （20分） \n",
    "3. 实现基于模型（矩阵分解）的协同过滤。（30分） \n",
    "4. 对每种推荐算法的推荐结果，用Top10个推荐歌曲的准确率和召回率评价推荐系统的性能。（30分） \n",
    "\n",
    "解题提示\n",
    "注意： \n",
    "1. 由于这个数据集中并没有用户对物品的显式打分，需要将播放次数转换为分数。 \n",
    "2. 在协同过滤中，计算用户之间的相似度或物品之间的相似度时，一种方式用播放次数/比例作为用户/物品的特征表示，同课件。 \n",
    "另一种可选的表示是只要用户播放过歌曲就表示为1，否则为0（二值化），这样物品之间的相似度为播放两个歌曲的用户交集除以播放两个歌曲的用户并集： \n",
    "。 \n",
    "类似的，两个用户之间的相似度可用两个用户播放歌曲的交集除以两个用户播放歌曲的并集表示。 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 这里时间关系，没有充分理解老师写的代码含义。。直接全部拿来使用"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "就导入的数据来源改了一下，函数传值改了一下，其他都没改，注释都没改。。。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "#load数据（用户和物品索引，以及倒排表）\n",
    "import _pickle as cPickle\n",
    "\n",
    "#稀疏矩阵，打分表\n",
    "import scipy.io as sio\n",
    "import os\n",
    "\n",
    "#距离\n",
    "import scipy.spatial.distance as ssd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#用户和item的索引\n",
    "users_index = cPickle.load(open(\"users_index.pkl\", 'rb'))\n",
    "items_index = cPickle.load(open(\"songs_index.pkl\", 'rb'))\n",
    "\n",
    "n_users = len(users_index)\n",
    "n_items = len(items_index)\n",
    "    \n",
    "#倒排表\n",
    "##每个用户打过分的电影\n",
    "user_items = cPickle.load(open(\"user_songs.pkl\", 'rb'))\n",
    "##对每个电影打过分的事用户\n",
    "item_users = cPickle.load(open(\"song_users.pkl\", 'rb'))\n",
    "\n",
    "#用户-物品关系矩阵R\n",
    "user_item_scores = sio.mmread(\"user_song_scores\")#.todense()\n",
    "user_item_scores = user_item_scores.tocsr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 计算每个用户的平均打分"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为这里有平均打分的计算，所以在之前的听歌次数转分数的时候，没有采用归一化，因为这里有，所以觉得没必要"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "users_mu = np.zeros(n_users)\n",
    "for u in range(n_users):  \n",
    "    n_user_items = 0\n",
    "    r_acc = 0.0\n",
    "    \n",
    "    for i in user_items[u]:  #用户打过分的item\n",
    "        r_acc += user_item_scores[u,i]\n",
    "        n_user_items += 1\n",
    " \n",
    "    users_mu[u] = r_acc/n_user_items"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 计算两个item之间的相似度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个函数的精髓步骤如下：\n",
    "1. <font color=red>找到两个用户打分对象的合集</font>\n",
    "2. 计算这两个用户对这个item各自的有效打分，即去掉各自的平均分\n",
    "3. 计算相似度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def item_similarity(iid1, iid2):\n",
    "    su={}  #有效item（两个用户均有打分的item）的集合\n",
    "    for user in item_users[iid1]:  #对iid1所有打过分的用户\n",
    "        if user in item_users[iid2]:  #如果该用户对iid2也打过分\n",
    "            su[user]=1  #该用户为一个有效user\n",
    "        \n",
    "    n=len(su)   #有效item数，有效item为即对uid对Item打过分，uid2也对Item打过分\n",
    "    if (n==0):  #没有共同打过分的item，相似度设为0？\n",
    "        similarity=0  \n",
    "        return similarity  \n",
    "        \n",
    "    #iid1的有效打分(减去用户的平均打分)\n",
    "    s1=np.array([user_item_scores[user,iid1]-users_mu[user] for user in su])\n",
    "        \n",
    "    #iid2的有效打分(减去用户的平均打分)\n",
    "    s2=np.array([user_item_scores[user,iid2]-users_mu[user] for user in su])  \n",
    "    \n",
    "    similarity = 1 - ssd.cosine(s1, s2) \n",
    "    if( np.isnan(similarity) ):  #分母为0（s1或s2中所有元素为0）\n",
    "        similarity = 0.0\n",
    "    return similarity  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 预计算好所有item之间的相似性\n",
    "对item比较少、Item比较固定的系统适用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "i=0 \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\15067\\Anaconda3\\lib\\site-packages\\scipy\\spatial\\distance.py:698: RuntimeWarning: invalid value encountered in double_scalars\n",
      "  dist = 1.0 - uv / np.sqrt(uu * vv)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "i=100 \n",
      "i=200 \n",
      "i=300 \n",
      "i=400 \n",
      "i=500 \n",
      "i=600 \n",
      "i=700 \n"
     ]
    }
   ],
   "source": [
    "items_similarity_matrix = np.matrix(np.zeros(shape=(n_items, n_items)), float)\n",
    "\n",
    "for i in range(n_items):\n",
    "    items_similarity_matrix[i,i] = 1.0\n",
    "    \n",
    "    #打印进度条\n",
    "    if(i % 100 == 0):\n",
    "        print (\"i=%d \" % (i) )\n",
    "\n",
    "    for j in range(i+1,n_items):   #items by user \n",
    "        items_similarity_matrix[j,i] = item_similarity(i, j)\n",
    "        items_similarity_matrix[i,j] = items_similarity_matrix[j,i]\n",
    "\n",
    "cPickle.dump(items_similarity_matrix, open(\"songs_similarity.pkl\", 'wb')) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 测试"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=red>这里不是很懂老师代码里的3个预测打分函数的区别。。。也不知道题目要求里的   “对每种推荐算法的推荐结果，用Top10个推荐歌曲的准确率和召回率评价推荐系统的性能”</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 预测用户对item的预测打分2\n",
    "\n",
    "利用用户打过分的item中，与item最相似的n_Knns最物品计算预测打分\n",
    "RMSE为1.13\n",
    "PR和覆盖度性能最好"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "### 预测用户对item的打分, 取该用户n_Knns最相似的物品\n",
    "def Item_CF_pred2(uid, iid, n_Knns): \n",
    "    sim_accumulate=0.0  \n",
    "    rat_acc=0.0 \n",
    "    n_nn_items = 0\n",
    "    \n",
    "    #相似度排序\n",
    "    cur_items_similarity = np.array(items_similarity_matrix[iid,:])\n",
    "    cur_items_similarity = cur_items_similarity.flatten()\n",
    "    sort_index = sorted(((e,i) for i,e in enumerate(list(cur_items_similarity))), reverse=True)\n",
    "    \n",
    "    for i in range(0,len(sort_index)):\n",
    "        cur_item_index = sort_index[i][1]\n",
    "        \n",
    "        if n_nn_items >= n_Knns:  #相似的items已经足够多（>n_Knns）\n",
    "            break;\n",
    "        \n",
    "        if cur_item_index in user_items[uid]: #对用户打过分的item\n",
    "           #计算当前用户打过分item与其他item之间的相似度\n",
    "            #sim = item_similarity(cur_item_index, iid)\n",
    "            sim = items_similarity_matrix[iid, cur_item_index]\n",
    "            \n",
    "            if sim != 0: \n",
    "                rat_acc += sim * (user_item_scores[uid, cur_item_index])   #用户user对item i的打分\n",
    "                sim_accumulate += np.abs(sim)  \n",
    "        \n",
    "            n_nn_items += 1\n",
    "        \n",
    "    if sim_accumulate != 0:  \n",
    "        score = rat_acc/sim_accumulate\n",
    "    else:   #no similar items,return average rates of the user   \n",
    "        score = users_mu[uid]\n",
    "    \n",
    "    if score <0:\n",
    "        score = 0.0\n",
    "    \n",
    "    return score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "#user：用户\n",
    "#返回推荐items及其打分（DataFrame）\n",
    "def recommend(user):\n",
    "    cur_user_id = users_index[user]\n",
    "    \n",
    "    #训练集中该用户打过分的item\n",
    "    cur_user_items = user_items[cur_user_id]\n",
    "\n",
    "    #该用户对所有item的打分\n",
    "    user_items_scores = np.zeros(n_items)\n",
    "\n",
    "    #预测打分\n",
    "    for i in range(n_items):  # all items \n",
    "        if i not in cur_user_items: #训练集中没打过分\n",
    "            user_items_scores[i] = Item_CF_pred2(cur_user_id, i, 10)  #预测打分\n",
    "    \n",
    "    #推荐\n",
    "    #Sort the indices of user_item_scores based upon their value，Also maintain the corresponding score\n",
    "    sort_index = sorted(((e,i) for i,e in enumerate(list(user_items_scores))), reverse=True)\n",
    "    \n",
    "    #Create a dataframe from the following\n",
    "    columns = ['song', 'score']\n",
    "    df = pd.DataFrame(columns=columns)\n",
    "         \n",
    "    #Fill the dataframe with top 20 (n_rec_items) item based recommendations\n",
    "    #sort_index = sort_index[0:n_rec_items]\n",
    "    #Fill the dataframe with all items based recommendations\n",
    "    for i in range(0,len(sort_index)):\n",
    "        cur_item_index = sort_index[i][1] \n",
    "        cur_item = list (items_index.keys()) [list (items_index.values()).index (cur_item_index)]\n",
    "            \n",
    "        if ~np.isnan(sort_index[i][0]) and cur_item_index not in cur_user_items:\n",
    "            df.loc[len(df)]=[cur_item, sort_index[i][0]]\n",
    "    \n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 读取测试数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "#读取测试数据\n",
    "dpath='./Data/'\n",
    "df_triplet_test= pd.read_csv(dpath +'RS_Song_test.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 测试，并计算评价指标\n",
    "PR、覆盖度、RMSE\n",
    "这部分代码所有的推荐算法相同\n",
    "\n",
    "令系统的用户集合为 U， R(u) 是根据用户在训练集上的行为给用户作出的推荐列表，而 T(u) 是用户在测试集上的行为列表。那么推荐结果的准确率定义为：\n",
    "$$\n",
    "Precision=\\frac{\\sum_{u\\in U}|R(u)\\cap T(u)|}{\\sum_{u\\in U}|R(u)|}\n",
    "$$\n",
    "推荐结果的召回率定义为：\n",
    "$$\n",
    "Recall=\\frac{\\sum_{u\\in U}|R(u)\\cap T(u)|}{\\sum_{u\\in U}|T(u)|}\n",
    "$$\n",
    "\n",
    "推荐系统的覆盖率为：\n",
    "$$\n",
    "Coverage=\\frac{\\sum_{u\\in U}|R(u)|}{|I|}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4e11f45d732f4861772b2906f81a7d384552ad12 is a new user.\n",
      "\n",
      "da681f423e8574d2581534fd138eae264dea0f5c is a new user.\n",
      "\n",
      "371e9aff6b9f644e7598dbc4dd5640b1544b97f3 is a new user.\n",
      "\n",
      "c0ff0f1c93f67c1fb372b36b1b08bb4c76bead7d is a new user.\n",
      "\n",
      "13ce57b3a25ef63fa614335fd838e8024c42ec17 is a new user.\n",
      "\n",
      "6a944bfe30ae8d6b873139e8305ae131f1607d5f is a new user.\n",
      "\n",
      "45f06d9d15adbd8575a63e848b1f1c202afbf308 is a new user.\n",
      "\n",
      "9acce6b22c8ae6adbd9e0933fb0a374b149e8e88 is a new user.\n",
      "\n",
      "283882c3d18ff2ad0e17124002ec02b847d06e9a is a new user.\n",
      "\n",
      "ee7aa84c164038c963cfd02a7e52a5598aa470c3 is a new user.\n",
      "\n",
      "b048f21afd5e7467f187bf9f9d413e97c32313a9 is a new user.\n",
      "\n",
      "04043d5716a2359f49f62d13c7c3d3e72b28f520 is a new user.\n",
      "\n",
      "7e40c676afe9ffcb80dfcb62e2acffd26ae21996 is a new user.\n",
      "\n",
      "ee30810179c611d32705fe0b71333dcb8703b30a is a new user.\n",
      "\n",
      "67874d1a189c83326c529e554be6f7acf55effae is a new user.\n",
      "\n",
      "e5b4f068ec1446a74564b4b8b8d13dcea7d27840 is a new user.\n",
      "\n",
      "1aa4fd215aadb160965110ed8a829745cde319eb is a new user.\n",
      "\n",
      "0091e0326c4c034cc04be6454742912845740a1f is a new user.\n",
      "\n",
      "30cc99a1b03a19cc162b286c76047e58371ffb3d is a new user.\n",
      "\n",
      "6ccd111af9b4baa497aacd6d1863cbf5a141acc6 is a new user.\n",
      "\n",
      "91ee487b65406cec50462071a41a83b5d6e99342 is a new user.\n",
      "\n",
      "2231cb435771a1a621ec44e95cdd28b81fad3288 is a new user.\n",
      "\n",
      "abaf265b7dc7d19732cde4e55e04a1ae65514e03 is a new user.\n",
      "\n",
      "4be63160ad51eb4c19c3ba70bf097b729d929650 is a new user.\n",
      "\n",
      "358e0fdc1b9d9a219ccea844b58bd8e6e35f0533 is a new user.\n",
      "\n",
      "6d625c6557df84b60d90426c0116138b617b9449 is a new user.\n",
      "\n",
      "330a7b1168a732742b572ba38a186c5b951b7f2d is a new user.\n",
      "\n",
      "81f0878d4707a39b4aecb412662f901e706adb35 is a new user.\n",
      "\n",
      "083a2a59603a605275107c00812a811526c2a0af is a new user.\n",
      "\n",
      "9c859962257112ad523f1d3c121d35191daa6d2b is a new user.\n",
      "\n",
      "dc02d3c4fb534ca1406f720c70678e5858a5ca0a is a new user.\n",
      "\n",
      "95926d03f275f4a3f04f068c2015e101a93a9292 is a new user.\n",
      "\n",
      "53f5a96ce4fe0b735815c877e225225e95061b1b is a new user.\n",
      "\n",
      "d9e5124935aebeb6b9b169df2d28e1fc658d3566 is a new user.\n",
      "\n",
      "debd74c1999b195eeddf09bc0af751c93a52c205 is a new user.\n",
      "\n",
      "7e89a43dc229de4259b5a36053de255a6247a773 is a new user.\n",
      "\n",
      "43e241eaa4d079ab2bdbd30231695f5563583af4 is a new user.\n",
      "\n",
      "d13609d62db6df876d3cc388225478618bb7b912 is a new user.\n",
      "\n",
      "ed3e56f28c9b1f6dd18270cf3c29f764ae8e4712 is a new user.\n",
      "\n",
      "7e41241058af7545be75fb1e38504520e88a0ceb is a new user.\n",
      "\n",
      "48567d388c6a7dda0e9d0a7b6648bdb42440475c is a new user.\n",
      "\n",
      "82cd7de99ecda208dcd18346892859f35daf0520 is a new user.\n",
      "\n",
      "360dfc77809fc2f4a5863eb59bcf4191de6635ef is a new user.\n",
      "\n",
      "dd29009e6148c8c21544c27e7c074b4601ba54e5 is a new user.\n",
      "\n",
      "b055cc3ae8159fe4bc656dc5631cc6659fc5cf72 is a new user.\n",
      "\n",
      "5ea608df0357ec4fda191cb9316fe8e6e65e3777 is a new user.\n",
      "\n",
      "a15075a926c1998d91940f118342ba8356efc7d4 is a new user.\n",
      "\n",
      "0b19fe0fad7ca85693846f7dad047c449784647e is a new user.\n",
      "\n",
      "70caceccaa745b6f7bc2898a154538eb1ada4d5a is a new user.\n",
      "\n",
      "fe2d77de7e57f3b3eedcf473545110b13ca03426 is a new user.\n",
      "\n",
      "4c6426c9c734c23ec0c35154437fc132b862068c is a new user.\n",
      "\n",
      "1820cfffd52cad7b3af398f379524d51579655d2 is a new user.\n",
      "\n",
      "972cce803aa7beceaa7d0039e4c7c0ff097e4d55 is a new user.\n",
      "\n",
      "d8aace9308e530e0f1ef2a85258b62f8245be423 is a new user.\n",
      "\n",
      "da3890400751de76f0f05ef0e93aa1cd898e7dbc is a new user.\n",
      "\n",
      "0a4c3c6999c74af7d8a44e96b44bf64e513c0f8b is a new user.\n",
      "\n",
      "5e53fe2ce11639e43554127e893dfe3b1b8cb1be is a new user.\n",
      "\n",
      "b85e44249a1d26a378c0757442452525ecfa1abf is a new user.\n",
      "\n",
      "c012ec364329bb08cbe3e62fe76db31f8c5d8ec3 is a new user.\n",
      "\n",
      "8c800d6ece52acb6aff6ab60d991d090c84aabdc is a new user.\n",
      "\n",
      "87024dfca766888f3eb47b2f1b71e2eff60ddb86 is a new user.\n",
      "\n",
      "11089077167c838f036384933d6c0ecafce6814b is a new user.\n",
      "\n",
      "4be305e02f4e72dad1b8ac78e630403543bab994 is a new user.\n",
      "\n",
      "952130c8ae0527458c75471812dbb5c506b46d5d is a new user.\n",
      "\n",
      "315213e4a0e89abdd69633c46f83b6e1dc0e8d87 is a new user.\n",
      "\n",
      "fef771ab021c200187a419f5e55311390f850a50 is a new user.\n",
      "\n",
      "25f83a09541073b1bde62e6e513cb938d3b148de is a new user.\n",
      "\n",
      "94d3dcdd520f1f2f1daec98d1f3736bcff751654 is a new user.\n",
      "\n",
      "54d07c4eff369f457c7a5d1c0b9b6c48feb949f4 is a new user.\n",
      "\n",
      "bb0d8c40d8932c5960e8e4443aa391adb9dfe9da is a new user.\n",
      "\n",
      "71dac99494dd7439a105d14f7597ae706b19c9ed is a new user.\n",
      "\n",
      "96f7b4f800cafef33eae71a6bc44f7139f63cd7a is a new user.\n",
      "\n",
      "2102cfa15b3702123d179c1559717e1886170139 is a new user.\n",
      "\n",
      "6a58f480d522814c087fd3f8c77b3f32bb161f9d is a new user.\n",
      "\n",
      "bb279e21a433803eb4a3dc5d31d2fed80d06fc98 is a new user.\n",
      "\n",
      "fc05f377863a77d7784b02de2cc06cdecb85968b is a new user.\n",
      "\n",
      "e5b1a35349132afe4f96dd6b39cdc4d94031eb1b is a new user.\n",
      "\n",
      "b7948bd0aab3932852dce33f89604a85736a449a is a new user.\n",
      "\n",
      "1fbb0f6de552dec6eff67288fbaf3efeae2a2427 is a new user.\n",
      "\n",
      "dae0bbb6e964a8f5495fa6edfb7321437b28b5a1 is a new user.\n",
      "\n",
      "7b01590cce3cb5c11efb729f55551b9ce252a00d is a new user.\n",
      "\n",
      "65c23b32ecf7636bd62af8f928967cd0f23b00d8 is a new user.\n",
      "\n",
      "c1912062175dc4b3ea5a3a0cdb963c704bb9c881 is a new user.\n",
      "\n",
      "6b2787bcf47cfea75b734729cdaddd700137ad5a is a new user.\n",
      "\n",
      "63aa3e991a33c309016051541b8084ff0aec7284 is a new user.\n",
      "\n",
      "c15570ec9ccc9f4093fc51dba952de0558c74ae7 is a new user.\n",
      "\n",
      "b5186694bb2406962edb3a8042b535449627fa37 is a new user.\n",
      "\n",
      "79a1ba30cec2ee1509f3c2df697916fe901b37e9 is a new user.\n",
      "\n",
      "f6ae5e682750e815c1709ca99138d03b039839d6 is a new user.\n",
      "\n",
      "e0152edeec1154b620b083fbb095f1f0f4de5e2e is a new user.\n",
      "\n",
      "edbeebbe98aeceb8d38b1de33ac1b1e201041284 is a new user.\n",
      "\n",
      "23ec7765590efc05002ccf6551cbadaa953c2069 is a new user.\n",
      "\n",
      "f8544ba8ff908f44d61f5d9d17c213423c1fc782 is a new user.\n",
      "\n",
      "2b04ee89ef77af7d11abf3077e366d1599656232 is a new user.\n",
      "\n",
      "de32503addf63b2ca79e523770bc9c790c3b4d63 is a new user.\n",
      "\n",
      "1d36c586adf56c1be114fb173bf23ececa79e9c7 is a new user.\n",
      "\n",
      "814dda72451239575dc8675ad72e0b612a3bba5e is a new user.\n",
      "\n",
      "e0314b2a3e48610bedb47f4663916170fb6188aa is a new user.\n",
      "\n",
      "f36597bc11ce99d965f4db2e065cf9d648bda39e is a new user.\n",
      "\n",
      "82b152c09a94dd4f7d4566080a00be441e3efbd9 is a new user.\n",
      "\n",
      "5d5a27d2fe3d45e515e79ddb7f70822ef5cd3600 is a new user.\n",
      "\n",
      "cbbe74dbde73f3b0005f9f1422d9f1e62cb54115 is a new user.\n",
      "\n",
      "7c9b8125c230d20b19317aa06a4b3f8415057f88 is a new user.\n",
      "\n",
      "e3fce0e4f316567685119c9af7d04d67c84f0a46 is a new user.\n",
      "\n",
      "05be6a588cc454d1e400a358c562807aeec8c054 is a new user.\n",
      "\n",
      "1198e6c8673f603114e7de4b1045a6cc6e80c897 is a new user.\n",
      "\n",
      "8ad6a386e7b601c0072c337ec6f86d084d1ba556 is a new user.\n",
      "\n",
      "4bcc4cfd9acf7e19bbccd398f8503ba79fb66513 is a new user.\n",
      "\n",
      "a4493d7bb8c07dc19e4208c8ed6925ec40300d36 is a new user.\n",
      "\n",
      "2f72cbcbf49aac438f614626d86c5e93efb82206 is a new user.\n",
      "\n",
      "9af7bb657715f4b8bb63588fc8c8a3ec46cbdb6e is a new user.\n",
      "\n",
      "3a6553c25f5dce3dbbbedc429496d40bf913e354 is a new user.\n",
      "\n",
      "7ffc14a55b6256c9fa73fc5c5761d210deb7f738 is a new user.\n",
      "\n",
      "6086417a706e339f64f47752b474e9c9596ff66d is a new user.\n",
      "\n",
      "2b77760cb7cf72d64dd6d5bdf8a5c195a044a9b3 is a new user.\n",
      "\n",
      "0084ecc8a2b3b0a371b968ded8b92d3d8525fd64 is a new user.\n",
      "\n",
      "d91ca966f809ef4a77999b031c90fcd5253fecc0 is a new user.\n",
      "\n",
      "2bc9df66877af6eddb7103859a574615f0a8aa91 is a new user.\n",
      "\n",
      "8ce501388bc1fa1c67f7563d9b2e13ce4554e375 is a new user.\n",
      "\n",
      "82fed477acd885cc428c29f1423f1929582d0643 is a new user.\n",
      "\n",
      "0c68affd09224b5231f656c31aec29505da2f661 is a new user.\n",
      "\n",
      "f3825861e5575cf4206f883630754fa30c80721e is a new user.\n",
      "\n",
      "e3dc1a577d1432b6406f3bed18d92fd6830ed69e is a new user.\n",
      "\n",
      "3f1037fb3a74f351cd61fff169a54896d2be06f9 is a new user.\n",
      "\n",
      "e3d45fa4071c85b044191cabf0284e74c744a037 is a new user.\n",
      "\n",
      "ffe5ad43c24d81878621185e164043a6e49b2fe4 is a new user.\n",
      "\n",
      "ffe2ec5b72cddb8537ad7f0ac191624f8ae2c8dc is a new user.\n",
      "\n",
      "eaa2b3c9e086a662ab15e10ca6211a3207f40a50 is a new user.\n",
      "\n",
      "69963f38f75cedf4029510ba20351fec7bdaede9 is a new user.\n",
      "\n",
      "d446e7c921a102071026003e6baf3175afc535e8 is a new user.\n",
      "\n",
      "cd58f3d02bfa9e5df95971cde778593012241236 is a new user.\n",
      "\n",
      "1a6916b63039c3c56c62de345c284c11a075eb96 is a new user.\n",
      "\n",
      "b6294fa6a76a84789d038b7133ddc308391bacc6 is a new user.\n",
      "\n",
      "9c6ebf1d5ba38bcb577149b22e19448f655f6252 is a new user.\n",
      "\n",
      "32e46bf89e523d8261aab60b31a766a4d24af83e is a new user.\n",
      "\n",
      "6049f073c9b55e1f0995902414dd64f41357ffa3 is a new user.\n",
      "\n",
      "d2396f84238693721349e3dfb9e109bb67b93fca is a new user.\n",
      "\n",
      "ab8fcf84d35479e7a2c88416040e67a78217fbda is a new user.\n",
      "\n",
      "b3dd0cfd6801e5af3a1a42acd9055f97c0112ad8 is a new user.\n",
      "\n",
      "ae552344f08d59c11b0a0b217f66b17ba9b37ba9 is a new user.\n",
      "\n",
      "821475e83c0f0cadfe0a30c3eb8034ff5c45f3af is a new user.\n",
      "\n",
      "c9ccbdf63666eb4b570bb9103f371346e2c3b378 is a new user.\n",
      "\n",
      "8cb51abc6bf8ea29341cb070fe1e1af5e4c3ffcc is a new user.\n",
      "\n",
      "2fa58dbe45db621eadad025c7bf3e30957eee321 is a new user.\n",
      "\n",
      "ae30f0628af54f394bc94e29c934913d1ef7cff6 is a new user.\n",
      "\n",
      "e8f6a8d06b0096737dec1a9f44c3d48cd9e5e4b8 is a new user.\n",
      "\n",
      "39797b8781fee9c03a218afddda0a4759dabc937 is a new user.\n",
      "\n",
      "2d1d1596c8d162b33ce8bdb575dae8073afa1086 is a new user.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#统计总的用户\n",
    "unique_users_test = df_triplet_test['user'].unique()\n",
    "\n",
    "#为每个用户推荐的item的数目\n",
    "n_rec_items = 10\n",
    "\n",
    "#性能评价参数初始化，用户计算Percison和Recall\n",
    "n_hits = 0\n",
    "n_total_rec_items = 0\n",
    "n_test_items = 0\n",
    "\n",
    "#所有被推荐商品的集合（对不同用户），用于计算覆盖度\n",
    "all_rec_items = set()\n",
    "\n",
    "#残差平方和，用与计算RMSE\n",
    "rss_test = 0.0\n",
    "\n",
    "#对每个测试用户\n",
    "for user in unique_users_test:\n",
    "    #测试集中该用户打过分的电影（用于计算评价指标的真实值）\n",
    "    if user not in users_index:   #user在训练集中没有出现过，新用户不能用协同过滤\n",
    "        print(str(user) + ' is a new user.\\n')\n",
    "        continue\n",
    "   \n",
    "    user_records_test= df_triplet_test[df_triplet_test.user == user]\n",
    "    \n",
    "    #对每个测试用户，计算该用户对训练集中未出现过的商品的打分，并基于该打分进行推荐（top n_rec_items）\n",
    "    #返回结果为DataFrame\n",
    "    rec_items = recommend(user)\n",
    "    \n",
    "    for i in range(n_rec_items):\n",
    "        item = rec_items.iloc[i]['song']\n",
    "        \n",
    "        if item in user_records_test['song'].values:\n",
    "            n_hits += 1\n",
    "        all_rec_items.add(item)\n",
    "    \n",
    "    #计算rmse\n",
    "    for i in range(user_records_test.shape[0]):\n",
    "        item = user_records_test.iloc[i]['song']\n",
    "        score = user_records_test.iloc[i]['rate']\n",
    "        \n",
    "        df1 = rec_items[rec_items.song== item]\n",
    "        if(df1.shape[0] == 0): #item在训练集中没有出现过，新item不能被协同过滤推荐\n",
    "            print(str(item) + ' is a new song.\\n')\n",
    "            continue\n",
    "        pred_score = df1['score'].values[0]\n",
    "        rss_test += (pred_score - score)**2     #残差平方和\n",
    "    \n",
    "    #推荐的item总数\n",
    "    n_total_rec_items += n_rec_items\n",
    "    \n",
    "    #真实item的总数\n",
    "    n_test_items += user_records_test.shape[0]\n",
    "\n",
    "#Precision & Recall\n",
    "precision = n_hits / (1.0*n_total_rec_items)\n",
    "recall = n_hits / (1.0*n_test_items)\n",
    "\n",
    "#覆盖度：推荐商品占总需要推荐商品的比例\n",
    "coverage = len(all_rec_items) / (1.0* n_items)\n",
    "\n",
    "#打分的均方误差\n",
    "rmse=np.sqrt(rss_test / df_triplet_test.shape[0])  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.08333333333333333"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0125"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coverage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.22579479795874388"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=red>最后结果精确度只有10%，这个原因在哪里呢？</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可能的原因：\n",
    "1. 次数转分数的时候给的分数阶梯太大了，1-15\n",
    "（这反面是否可以说，如果想让模型的精确度好看一点，我其实可以转成分数的时候粗矿一点，直接打分成1-3，这样预测命中的概率应该很大吧，但是这个现实生活中似乎失去了意义？）\n",
    "2. 还是次数转分数的方案不好，但是不知道怎么个不好，感觉不好\n",
    "3. 存在脏数据？对于一些用户只听了几首歌的，是否没有训练的价值？应该不是的吧\n",
    "4. 数据量太少，3w7k多条数据，790个用户，800首歌，最多的用户听过25%的歌，其实也算是稀疏矩阵了，可以这么理解吗？"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
